Pre-training of neural network by parameter decomposition

ABSTRACT

A technique for training a neural network including an input layer, one or more hidden layers and an output layer, in which the trained neural network can be used to perform a task such as speech recognition. In the technique, a base of the neural network having at least a pre-trained hidden layer is prepared. A parameter set associated with one pre-trained hidden layer in the neural network is decomposed into a plurality of new parameter sets. The number of hidden layers in the neural network is increased by using the plurality of the new parameter sets. Pre-training for the neural network is performed.

BACKGROUND OF THE INVENTION

The present disclosure, generally, relates to machine learningtechniques, and more particularly, to technique for training a neuralnetwork including an input layer, one or more hidden layers and anoutput layer.

It has been known that pre-training neural networks before fine-tuningcan improve automatic speech recognition (ASR) performance, especiallywhen the amount of training data is relatively small. There are severalknown pre-training techniques including discriminative pre-training thatis based on an error back-propagation and generative pre-training thatdoes not use discriminative information.

In conventional pre-training processes, a new layer initialized withrandom parameters is inserted to the top of the hidden layers just belowthe output layer. Then, the neural network is pre-trained using thetraining data.

SUMMARY

According to an embodiment of the present invention there is provided acomputer-implemented method for training a neural network that includesan input layer, one or more hidden layers and an output layer. Themethod includes preparing a base of neural network, which includes atleast a pre-trained hidden layer. The method also includes decomposing aparameter set associated with one pre-trained hidden layer in the neuralnetwork into a plurality of new parameter sets. The method furtherincludes increasing the number of the hidden layers in the neuralnetwork based at least in part on the plurality of the new parametersets. The method includes further performing pre-training for the neuralnetwork.

Computer systems and computer program products relating to one or moreaspects of the present invention are also described and claimed herein.

According to other embodiment of the present invention there is provideda computer-implemented method for training a neural network thatincludes an input layer, one or more hidden layers and an output layer.The method includes preparing a base of the neural network. The methodalso includes repeatedly performing insertion of a new hidden layer intothe neural network and pre-training for the neural network until thenumber of the hidden layers in the neural network reaches apredetermined number while a position to be inserted the new hiddenlayer is identical to a position having been inserted in a previousstage.

Additional features and advantages are realized through the techniquesof the present invention. Other embodiments and aspects of the inventionare described in detail herein and are considered a part of the claimedinvention.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter, which is regarded as the invention, is particularlypointed out and distinctly claimed in the claims at the conclusion ofthe specification. The forgoing and other features and advantages of theinvention are apparent from the following detailed description taken inconjunction with the accompanying drawings in which:

FIG. 1 illustrates a block diagram of a speech recognition system thatincludes an acoustic training system for training a neural network basedacoustic model in accordance with at least one embodiment of the presentinvention;

FIG. 2 depicts an architecture of a neural network used for an acousticmodel, which is a target of a novel training function in accordance withat least one embodiment of the present invention;

FIG. 3 describes a standard way of inserting a new hidden layer into aneural network during pre-training process in accordance with at leastone embodiment of the present invention;

FIG. 4 illustrates a more detailed block diagram of a pre-trainingprocessing module in accordance with at least one embodiment of thepresent invention;

FIG. 5 describes a way of inserting new hidden layers into a neuralnetwork in accordance with at least one embodiment of the presentinvention;

FIG. 6 is a flowchart depicting a novel training process for training aneural network based acoustic model in accordance with at least oneembodiment of the present invention; and

FIG. 7 depicts a schematic of a computer system in accordance with atleast one embodiment of the present invention.

DETAILED DESCRIPTION

Now, the present invention will be described using particularembodiments, and the embodiments described hereafter are understood tobe only referred to as examples and are not intended to limit the scopeof the present invention.

One or more embodiments according to the present invention are directedto computer-implemented methods, computer systems and computer programproducts for training a neural network including an input layer, one ormore hidden layers and an output layer, in which the trained neuralnetwork can be used for a task such as speech recognition.

Referring to the series of FIGS. 1-6, there are shown a computer systemand a method for training an acoustic model based on a neural networkaccording to some embodiments of the present invention.

Referring to FIG. 1, a block diagram of a speech recognition system thatincludes an acoustic model training system in accordance with at leastone embodiment of the present invention is illustrated. As shown in FIG.1, the speech recognition system 100 may include an acoustic featureextractor 102 that receives speech signals and extracts acousticfeatures from the received speech signals; a speech recognition engine104 that receives the extracted acoustic features and outputs a decodedresult based on given speech recognition models 106; and an acousticmodel training system 120.

The acoustic feature extractor 102 receives the speech signalsdigitalized by sampling analog audio input, which may be input from amicrophone for instance, at a predetermined sampling frequency and apredetermined bit depth. The acoustic feature extractor 102 extracts theacoustic features from the received speech signal by any known acousticfeature analysis and then outputs a sequence of the extracted acousticfeatures. The speech signal may be provided as an audio file, an audiostream from input device such as microphone, or an audio stream vianetwork socket.

In one or more embodiments, the acoustic features may include, but notlimited to, MFCC (Mel Frequency Cepstral Coefficient), LPC (LinearPredictive Coding) Coefficient, PLP (Perceptual Liner Prediction)Cepstral Coefficient, log Mel spectrum, raw input features, or anycombinations thereof. The acoustic features may further includedynamical features such as delta features and delta-delta features ofthe aforementioned acoustic features.

The speech recognition engine 104 receives the sequence of the extractedacoustic features and predicts most plausible speech contents based onthe speech recognition models 106.

The speech recognition models 106 may include a language model 108 andan acoustic model 110. The language model 108 is a model representingprobability distribution of word sequence and may be, but not limitedto, n-gram model or neural network based model. The acoustic model 110is a model representing relationship between input acoustic features andlinguistic units constituting a speech. Among the speech recognitionmodels 106, the acoustic model 110 may be a target of the novel trainingfunction according at least one embodiment of the present invention. Theacoustic model 110 will be described in more detail later.

The speech recognition engine 104 finds a word sequence with maximumlikelihood based on the sequence of the acoustic features provided fromthe acoustic feature extractor 102 by integrating the language model 108and the acoustic model 110, and outputs the word sequence found as thedecoded result.

The acoustic model training system 120 performs the training processaccording to an embodiment of the present invention to build theacoustic model 110. In some embodiments, the acoustic model 110 is aneural network based model that includes a neural network having aninput layer, one or more hidden layers and an output layer.

The acoustic model training system 120 uses a given set of training datato build the acoustic model 110. In some embodiments, speech data 150with transcription is provided as the set of the training data. Thespeech data 150 include a plurality of utterances and correspondingtranscriptions. The speech data 150 may be stored in a storage deviceoperably coupled to a processing circuitry.

In some embodiments, a two-steps training process including pre-trainingprocess and fine-tuning process is conducted by using the given set ofthe training data. For this purpose, the acoustic model training system120 may include a pre-training processing module 130 for performingpre-training process that builds up the neural network with parametersin layer-by-layer manner; and a fine-tuning processing module 140 fordiscriminatively updating parameters of the neural network based uponthe parameters given by the pre-training processing module 130.

Referring to FIG. 2, an architecture of a neural network, which can beused for the acoustic model 110 for speech recognition, is shown. Theneural network 160 shown in FIG. 2 is one example of such neural networkto which the novel training process can be applicable.

The neural network 160 depicted in FIG. 2 includes an input layer 162,one or more convolutional layers 164, one or more fully-connected layers166 and an output layer 168. The neural network 160 shown in FIG. 2 hasa typical configuration of CNN (Convolutional Neural Network), which isone of the most promising models used in a variety of recognitionprocessing such as speech recognitions, image recognitions, motionrecognitions, etc.

A plurality of consecutive feature frames may be used as input for theneural network 160. Each feature frame includes a plurality of featuresextracted from original speech signal by the acoustic feature extractor102. In a particular embodiment, the input for the neural network 160can be represented as two-dimensional space in time (e.g. frames) andfrequency (e.g. frequency band) axes as shown in FIG. 1. Also, the inputfor the neural network 160 can be represented by using further otherdimension such as feature types. For example, the number of featuretypes may be three when static features, delta feature, and delta-deltafeatures are used.

The one or more convolutional layers 164 may include one or more sets ofa convolutional layer followed by a subsampling layer. A hidden unit inthe convolutional layer takes inputs from a rectangular local windowspanning time and frequency axes (e.g. N frames ×M frequency bands,etc.) with one or more feature types (e.g. static/delta/delta-delta) inthe input layer 162 and, multiplies those local inputs by using alocalized filter (ϕ_(i)) that extracts local features from the input.The convolutional layer accommodates totally K feature maps generatedwith different localized filters ϕ_(i) (i=1, . . . , K), where K denotesthe number of localized filters. The subsampling layer takes inputs froma local region of the previous convolutional layer and down-samples theinputs with a subsampling operation, which may be, but not limited to,any one of max-pooling, average pooling and stochastic pooling.

The one or more fully-connected layers 166 take output of all units fromthe previous layer to perform class discrimination. The one or morefully-connected layers 166 may be made up of one or more hidden layerswith a plurality of hidden units, each of which has full connections toall units in the previous layer. The output layer 168 outputs posteriorprobability over targets corresponding to central frame while each inputof the neural network 160 is constituted from consecutive feature framescentering at time.

Note that states of HMMs, which may be mono-phone HMMs or multi-phoneHMMs, are used as the targets for class discrimination. The mono-phoneHMMs are context-independent models. On the other hand, the multi-phoneHMMs are context-dependent models. One of the models is a quin-phonemodel where each distinct phone model for every different two left andtwo right phone contexts is used.

The number of the sets in the one or more convolutional layers 164 andthe number of the hidden layers in the one or more fully-connectedlayers 166 may be set to appropriate numbers, respectively. The one ormore fully-connected layers 166 may or may not include a bottlenecklayer before the output layer 168.

The input is fed into the input layer 162 in the neural network 160 andthen the neural network 160 outputs result from the output layer 168. Inautomatic speech recognition systems (ASRs), the neural network 160 mayoutput observation probability over HMM states for input acousticfeatures.

Parameters of the neural network 160, which may include weights betweeneach units and biases of each unit, are optimized by training algorithmso as to classify the input correctly. Typically, the neural network 160are first subjected to pre-training process by the pre-trainingprocessing module 130 and then fine-tuning process by the fine-tuningprocessing module 140.

Referring to FIG. 3, a standard way of inserting a new hidden layer intoa neural network during a pre-training process is described. As shown inFIG. 3, in the standard pre-training process, generally, one new hiddenlayer is inserted with parameters that are initialized with randomvalues on a top of current structure of the neural network just below anoutput layer and pre-trained based upon the randomly initializedparameters.

However, there is no physical meaning in initializing parameters of thehidden layers with random values. Also, there is no physical meaning ininserting a new layer on top of the structure just below the outputlayer. Since the initial parameters of the neural network 160 may affecta result of subsequent training, there is room for improvement onparameter initialization before the subsequent training.

By referring to FIG. 4, a more detailed block diagram of thepre-training processing module 130 is described. As shown in FIG. 4, thepre-training processing module 130 may include a base preparation module132; a decomposition module 134; a layer replacement module 136 and apre-training module 138.

A set of training data are generated from the speech data 150 withtranscription via the acoustic feature extractor 102. Each training dataincludes acoustic features having the size of the local window andfeature types, and an associated class label. Any one of phonesincluding mono-phone HMM states and multi-phone HMM states or syllablemay be used as the class label.

The pre-training processing module 130 perform pre-training process thatfinds representation that well captures characteristics of input andstack layers with initial parameters for the subsequent fine-tuningprocess in a layer-by-layer manner.

The base preparation module 132 is configured to prepare a base of theneural network 160 as an initial condition for the pre-training process.The base of the neural network 160 may include the one or moreconvolutional layers 164 and one or more hidden layers corresponding toa part of the fully-connected layers 166. The one or more convolutionallayers 164 and the one or more hidden layers can be pre-trained by anyknown pre-training technique using the given set of the training data.Thus, the base of the neural network 160 prepared by the basepreparation module 132 may have at least one pre-trained hidden layercorresponding to one of the fully-connected layers 166.

The decomposition module 134 is configured to decompose a parameter setassociated with one pre-trained hidden layer 160 a in the neural network160 into a plurality of new parameter sets. One of the pre-trainedhidden layers in current structure of the neural network 160, whichcorresponds to one of the fully-connected layers 166, is targeted forparameter decomposition. The parameters associated with the onepre-trained hidden layer 160 a may include a weight matrix M associatedwith the one pre-trained hidden layer 160 a and a lower layer 160 bbelow the one pre-trained hidden layer 160 a and a bias vector Bassociated with the one pre-trained hidden layer 160 a. In a particularembodiment, the weight matrix M associated with the one pre-trainedhidden layer 160 a can be used as the parameter set to be decomposed bythe decomposition module 134.

In one or more embodiments, the parameter set can be decomposed byapplying any known matrix decomposition technique to a matrixrepresenting the parameter set. In a preferable embodiment, matrixdecomposition technique that decomposes the matrix into multipleorthogonal matrices can be employed. In a further preferable embodiment,the parameter set can be decomposed by applying singular valuedecomposition (SVD) to the matrix to generate a first orthogonal matrix,a singular value matrix and a second orthogonal matrix.

In the particular embodiment where the weight matrix M is used as theparameter set to be decomposed, the weight matrix M associated with theone pre-trained hidden layer 160 a can be decomposed as follows:M=UΣV ^(T),where M denotes an original weight matrix before decomposition, Udenotes a first orthogonal weight matrix after the SVD, Σ denotes asingular value matrix and V denotes a second orthogonal weight matrixafter the SVD.

The new parameter sets generated by the decomposition module 134 may bebased at least in part on the first orthogonal matrix U, the singularvalue matrix Σ and the second orthogonal matrix V. In a particularembodiment where the weight matrix M is used as the parameter set to bedecomposed, a first new parameter set of the plurality of the newparameter sets may have a new weight matrix that is the first orthogonalmatrix U, and a second new parameter set of the plurality of the newparameter sets may have a new weight matrix that is a matrix product ofthe singular value matrix and the transposed second orthogonal matrixΣV^(T).

The layer replacement module 136 is configured to increase the number ofthe hidden layers in the neural network 160 based at least in part onthe plurality of the obtained new parameter sets. The layer replacementmodule 136 may replace the one pre-trained hidden layer 160 a, whoseparameter has been decomposed by the decomposition module 134, withmultiple new hidden layers, more specifically two new hidden layers. Insome embodiments, the layer replacement module 136 may set the first andsecond new parameter sets to upper and lower layers of the two newhidden layers, respectively, as initial conditions for subsequentpre-training, in a manner such that the two new hidden layers becomesequivalent to (at least approximates) the original pre-trained hiddenlayer 160 a as expressed by the following equation:Mx=(UW)x=U(Wx),where W denotes ΣV^(T) and x denotes an input vector.

The first new parameter set U may be set as a weight matrix ofconnections between the upper and lower layers. The second new parameterset W(=ΣV^(T)) may be set as a weight matrix of connections between thelower layer and further lower hidden layer 106 b below the originalpre-trained hidden layer 106 a.

Note that if one pre-trained hidden layer 160 a connected to a lowerhidden layer 160 b having the same number of hidden units is targetedfor the parameter decomposition and the replacement, the one pre-trainedhidden layer 160 a may be replaced by two new hidden layers having thesame number of hidden units since the weight matrix M between two layersis regular and obtained orthogonal weight matrices U, V and the singularvalue matrix Σ are also regular. However, one pre-trained hidden layertargeted for the parameter decomposition and the replacement is notlimited to one connected to the lower hidden layer having the samenumber of hidden units. In other embodiments, one pre-trained hiddenlayer connected to a lower hidden layer having different number ofhidden units can be also targeted for the parameter decomposition andthe replacement.

In the particular embodiment where the weight matrix is used as theparameter set to be decomposed, new bias vectors can be calculated fromthe original bias vector B to set to the upper and lower layers. In aparticular embodiment, a new bias vector B₁ for the upper layer may bepreferably set to be identical to the original bias vector B of the onepre-trained hidden layer 160 a and a new bias vector B₂ for the lowerlayer may be set to be zero (B₁=B; B₂=0; TYPE 1).

Hereinabove, the particular embodiment where the weight matrix M is usedas the parameter set to be decomposed has been described. However, inother particular embodiment, both of the weight matrix M and the biasvector B can be used as the parameter set to be decomposed.

In the particular embodiment where the weight matrix M and the biasvector B are used as the parameter set, the matrix M′ to be decomposedby SVD can be expressed by using the weight matrix M with an extendedvector of the bias vector B with constant of 1 as follows:

$M^{\prime} = {\begin{bmatrix}M & B \\0 & 1\end{bmatrix}.}$

As shown in the aforementioned equation, the matrix M′ has a block ofthe weight matrix M with zeros and a block of an extended bias vectorarranged next to the block of the weight matrix M. The extended biasvector has a bias vector B of the one pre-trained hidden layer and aconstant of 1 attached thereto.

Assuming that W denotes ΣV^(T), the SVD applied to the weight matrix M′can be expressed by following block matrix form:

$\begin{matrix}{M^{\prime} = {U^{\prime}\Sigma^{\prime}V^{\prime\; T}}} \\{{= {{\begin{bmatrix}U & B_{1} \\0 & 1\end{bmatrix}\begin{bmatrix}\Sigma & 0 \\0 & 1\end{bmatrix}}\begin{bmatrix}V & B_{2} \\0 & 1\end{bmatrix}}},} \\{{= {\begin{bmatrix}U & B_{1} \\0 & 1\end{bmatrix}\begin{bmatrix}W & B_{2}^{\prime} \\0 & 1\end{bmatrix}}},} \\{{= \begin{bmatrix}{UW} & {{UB}_{2}^{\prime} + B_{1}} \\0 & 1\end{bmatrix}},}\end{matrix}$where B₂′ denotes ΣB₂.

By not considering the new bias vector B₁ and B₂, matrices U, Σ and Vcan be obtained. New weight matrices for the first and second newparameter sets may be U and ΣV^(T), respectively as similar to theaforementioned particular embodiment where the weight matrix M is usedas the parameter set. The bias vectors for the first and second newparameter sets may be obtained from original bias vector B, the firstorthogonal matrix U′, the singular value matrix Σ′ and/or the secondorthogonal matrix V′.

More specifically, considering given matrix M′, U B₂′+B₁ should be equalto the original bias vector B. If the new bias vector B₂′ for the lowerlayer is obtained from aforementioned equation, the new bias vector B₁for the upper layer may be set to be a value calculated from the biasvector B and the new bias vector B₂ for the lower layer as follows:B ₁ =B−UB′ ₂(TYPE 2).

The pre-training module 138 is configured to perform pre-training forthe neural network by using at least a part of the given set of thetraining data. Before the pre-training, the output layer 168 currentlyobtained after previous pre-training may be discarded and a new outputlayer is added. By performing the pre-training, at least the parametersets associated with the new multiple hidden layers are pre-trainedbased upon parameter sets given by the layer replacement module 136 asinitial conditions. In a particular embodiment, whole of the neuralnetwork may be pre-trained by using the given set of the training data.In some embodiments, the pre-training is discriminative pre-training, inwhich discriminative information, i.e., that is class label, is used forerror back-propagation procedure. However, in other embodiment, thepre-training is not limited to the discriminative pre-training, andgenerative pre-training, in which discriminative information are notused and reconstruction of input is performed, can also be employed.

The decomposition of the parameter set associated with one pre-trainedhidden layer 160 a by the decomposition module 134, the increase of thenumber of the hidden layer by the layer replacement module 136 and thepre-training for the neural network by pre-training module 138 may berepeatedly performed in a layer-by-layer manner until the number of thehidden layers (in the fully-connected layers 166) reaches apredetermined number.

FIG. 5 describes a way of inserting new hidden layers into the neuralnetwork. As show in FIG. 5, at first, a base 170 of the neural network160 is prepared by any known pre-training method. The base 170 shown inFIG. 5 includes two convolutional layers and two hidden layers, each ofwhich can be pre-trained by standard pre-training technique using thegiven set of the training data.

After given the base 170, a parameter set (M and/or B) associated withone hidden layer 170 a that is pre-trained up to the current stage isdecomposed into two new parameter sets (U and/or B₁ and ΣV^(T) and/orB₂). Then, the one hidden layer 170 a is replaced with two new hiddenlayers 172 a, 172 b to obtain a new neural network 172. The first newparameter set (U and/or B₁) is set to the upper layer 172 b and thesecond new parameter set (ΣV^(T) and/or B₂) is set to the lower layer172 a. In the neural network 172, structure below the lower layer 172 amay be copied from the base 170 of previous stage and a new output layerare added on a top of the upper layer 172 b. The neural network 172after replacement may be subjected to the pre-training to obtain a newbase 174 for next pre-training stage.

After given the new base 174, a parameter set (M and/or B) associatedwith one hidden layer 174 a pre-trained up to the current stage isdecomposed into two new parameter sets (U and/or B₁ and ΣV^(T) and/orB₂). Then the one hidden layer 174 a is replaced with two new hiddenlayers 176 a, 176 b. The first new parameter set (U and/or B₁) is set tothe upper layer 176 b and the second new parameter set (ΣV^(T) and/orB₂) is set to the lower layer 176 a. The neural network 176 afterreplacement may be subjected to the pre-training to obtain a further newbase 178 for next pre-training stage.

The procedure, which includes insertion of a new hidden layer into theneural network and a discriminative pre-training for the neural network,may be performed repeatedly until the number of the hidden layers in theneural network reaches a predetermined number.

In general, any one of the pre-trained hidden layers, which correspondsto one of the fully-connected layers 166, can be targeted for parameterdecomposition. However, in a preferable embodiment, during thelayer-by-layer pre-training process, a position to be decomposed and tobe inserted the new hidden layer are fixed to a positions having beendecomposed and inserted in a previous stage as shown in FIG. 5. In FIG.5, a position corresponding to the second fully-connected layer isalways targeted for the parameter decomposition and new layer insertionduring the whole pre-training process.

By inserting the new layers into the fixed position instead of insertingon a top hidden layer just below the output layer as done in standarddiscriminative pre-training, good quality error signals are expected tobe back propagated in back propagation procedure of the discriminativepre-training.

After the number of the hidden layers in the neural network reaches thepredetermined number, initial parameters of the whole neural network forsubsequent fine-tuning, which includes the weight matrices and the biasvectors for all layers, are obtained. Finally, the whole neural network160 is constructed by stacking the output layer 168 on the top hiddenlayers and by adding the connections between units of the top hiddenlayer and units of the output layer 168 with arbitrary parameters. Thewhole neural network 160 including the initial parameters and newlyadded arbitrary parameters are discriminatively tuned by updating theparameters using backpropagation algorithm during the supervisedfine-tuning process. In the fine-tuning process, a training parametersuch as a learning rate may be different from the discriminativepre-training. In some embodiments, the training process for the neuralnetwork 160 after the pre-training is not limited to the aforementionedtechniques, any known training technique can be applicable to the neuralnetwork 160.

In particular embodiments, each of modules 130, 140 of the acousticmodel training system 120 described in FIG. 1 as well as each ofsubmodules 132-138 of the pre-training processing module 130 describedin FIG. 4 may be implemented as, but is not limited to, the following:(i) a software module including program instructions and/or datastructures in conjunction with hardware components such as a processor,a memory, etc.; (ii) a hardware module including electronic circuitry;and/or (iii) a combination thereof. Modules 130, 132, 134, 136, 138 and140 described in FIG. 1 and FIG. 4 may be implemented on a singlecomputer system such as a personal computer and a server machine or acomputer system distributed over a plurality of devices such as acomputer cluster of the computer devices. Also, other components 102,104, 106, 150 may be implemented on the same computer system thatimplements modules 130, 132, 134, 136, 138 and 140 or on anothercomputer system.

The neural network 160 working as the acoustic model 110 is shownillustratively in FIG. 2. However, the neural network for the acousticmodel 110 can be based on any one of known neural network architecturesincluding other types of feedforward neural network such as a fullyconnected neural network, residual network, deep convolutional neuralnetworks developed by Oxford's Visual Geometry Group (VGG). Also, insome embodiments, the acoustic model 110 may be a hybrid NN (NeuralNetwork)-HMM model, where the neural network is used to directly computeobservation probability distribution for each HMM state instead ofstandard Gaussian Mixture Models (GMM) in the GMM/HMM system. However,the acoustic model 110 is not limited to the aforementioned NN-HMMmodel. In other embodiment, the acoustic model 110 may be other type ofNN-HMM model that is based on tandem or “bottleneck feature” approach,where the neural network is used to extract features as input for asubsequent system such as a standard GMM/HMM system and other neuralnetwork based system having different architecture or structure from theneural network in place of or in addition to standard acoustic features.

Referring to FIG. 6, a flowchart depicting novel training process fortraining a neural network-based acoustic model according to someembodiments of the present invention is shown. Note that thedecomposition of the parameter set is done by the SVD in the trainingprocess shown in FIG. 6.

As shown in FIG. 6, process may begin at step S100 in response toreceiving a request for initiating a training process from an operator.Note that the process shown in FIG. 6 may be performed by a processingcircuitry such as a processing unit that implements the acoustic modeltraining system 120 in FIG. 1, for the given speech data 150 stored in astorage device operably coupled to the processing unit.

At step S101, the processing unit may prepare a base of a neural network160 based on standard method. In step S101, the base of the neuralnetwork 160 may be prepared by performing pre-training of convolutionallayers 164 and pre-training of at least one hidden layer. At step S102,the processing unit may decompose a parameter set associated with onepre-trained hidden layer currently obtained in the neural network intotwo new parameter sets that is based on the orthogonal weight matricesU, V and the singular value matrix Σ. The two new parameter sets mayinclude a first new parameter set (U and/or B₁) and a second newparameter set (ΣV^(T) and/or B₂).

At step S103, the processing unit may replace the one pre-trained hiddenlayer with two new hidden layers. At step S104, the processing unit mayset the first new parameter set (U and/or B₁) to an upper layer of thetwo new hidden layers. At step S105, the processing unit may set thesecond new parameter set (ΣV^(T) and/or B₂) to a lower layer of the twonew hidden layers.

By the processing from the step S103 to the step 105, the number of thehidden layers in the neural network may be increased by one using thetwo new parameter sets. At step S106, the processing unit may performpre-training for the neural network 160, in which at least the two newhidden layers are pre-trained. In step S106, the discriminativepre-training, in which discriminative information is used for errorback-propagation procedure, can be preferably performed.

At step S107, the processing unit may determine whether the number ofthe hidden layers in the neural network 160 reaches a predeterminednumber or not. In response to determining that the number of the hiddenlayers does not reach the predetermined number (S107: NO), the processmay transfer to step S102 to proceed a next stage of pre-training.

In response to determining that the number of the hidden layers reachesthe predetermined number (S107: YES), the process may transfer to stepS108. At step S108, the processing unit may perform a fine-tuningprocess based on parameter initialized by the processing from the stepS101 to the step 107, in which the neural network is tuned bydiscriminatively updating parameters to predict target class. At stepS109, the processing unit may store the constructed neural network asthe acoustic model 110 and the process may end at step S110.

According to one or more embodiments of the present invention, bydecomposing a parameter set associated with one pre-trained hidden layerin the neural network into a plurality of new parameter sets and byincreasing the number of the hidden layers in the neural network basedat least in part on the plurality of the new parameter sets, a neuralnetwork with a desired depth can be efficiently built up and performanceof speech recognition by using the built neural network is expected tobe improved especially when the amount of training data is relativelysmall.

According to one or more embodiments of the present invention, byinserting the new layer into the fixed position, error signals areexpected to be back propagated through the neural network while keepingthe error signal with good quality in comparison with a standard waywhere the new layer having random parameters is inserted on a top hiddenlayer just below the output layer each time pre-training stageprogresses, which eases error back propagation in the discriminativepre-training.

The languages to which the novel training function may be applied arenot limited and include natural and designed human languages, computerlanguages, and the like.

Also note that in the embodiment having been described above, a neuralnetwork used for acoustic model 110 is described to be a target of thenovel training function. However, since a neural network is one of themost promising models used in a variety of recognition processing inaddition to speech recognition, the neural network to be trained by thenovel training function according to some embodiments may be a neuralnetwork used in other field such as image recognition processing, motionrecognition processing, etc.

Experimental Study

A program implementing the system and process shown in FIGS. 1, 4-6according to some embodiments was coded and executed for given trainingspeech samples. A neural network based acoustic mode (NN-AM) shown inFIG. 2 with convolutional and max pooling layers, fully-connected layersand input and output layers was created as a target of the trainingprocess.

The number of the convolutional layers was two. The numbers of thelocalized filters in the first and second convolutional layers were 128and 256, respectively. The fully-connected layers in the neural networkincluded 6 hidden layers of 1024 hidden units. The number of units inthe output layer of the neural network was 2000. Each unit in the outputlayer corresponded to each quin-phone HMM state.

15 hours of noisy speech data with manual transcriptions that isso-called Aurora-4 were prepared for training. 11 consecutive frames ofLog Mel features having 40 frequency banks and its dynamic features(delta and delta-delta features) were used as input as represented by[40×3×11]. A class label was aligned to each center frame by the forcedalignment technique based on standard GMM/HHM system.

As for comparative examples, the neural networks were trained bystandard backpropagation algorithm without discriminative pre-training(Comparative Example 1), standard discriminative pre-training withinserting a new layer initialized with random value on a top below theoutput layer (Comparative Example 2), respectively. As for examples(Example 1 and Example 2), the neural networks were trained by thetraining process shown in FIG. 6 with respective bias vectors (TYPE 1and TYPE 2). As for other example (Example 3), the neural network wastrained by the training process shown in FIG. 6 without decomposition,where the new layer initialized with random values was inserted to fixedposition each time pre-training stage progressed during the pre-trainingprocess.

In the examples and the comparative examples, after the training of theneural network was completed, the neural network from the input layer tothe output layer was stored as the acoustic model. The test speech dataincluding test data sets labeled with “clean” and “noisy” in theAurora-4 were used. ASR accuracy of the obtained speech recognitionmodels was evaluated for the examples and the comparative examples byusing several test data sets. WER (Word Error Rate) was utilized as ASRaccuracy metric.

Unless otherwise noted, any portions of the speech recognition modelexcept for way of training of the acoustic model were approximatelyidentical between the examples and the comparative examples. The finaltopologies of the neural network in the examples were identical to thatof the comparative examples. The subsequent fine-tuning after thepre-training in the examples (Example 1-3) was also identical to that ofthe comparative example 2.

The evaluated results of the examples and the comparative examples aresummarized as follows:

Test Condition wv2(mis- wv1(matched matched channel channel condition)condition) aver- Acoustic Model Clean Noisy Clean Noisy age ComparativeExample 1 4.61 8.08 7.58 16.35 11.34 (w/o discriminative pre-training)Comparative Example 2 4.54 8.20 6.93 15.40 10.96 (w/discriminativepre-training that inserts on a top hidden layer) Example 1 4.32 7.806.52 14.71 10.42 (w/discriminative pre-training that inserts a new layerinitialized with decomposed parameter set (including TYPE 1 bias) tofixed position) Example 2 4.46 7.55 6.44 14.79 10.35 (w/discriminativepre-training that inserts a new layer initialized with decomposedparameter set (including TYPE 2 bias) to fixed position) Example 3 4.457.91 6.82 15.04 10.64 (w/discriminative pre-training that inserts a newlayer initialized with random parameter set to fixed position)

The baseline NN-AMs showed 11.34% (Comparative Example 1) and 10.96%(Comparative Example 2) WER for average. By comparison with the resultof the comparative examples, WERs of all examples were shown to bereduced, indicating improvement of recognition accuracy. Even Example 3,where discriminative pre-training that merely inserts a new layerinitialized with random parameter set to a fixed position withoutdecomposition was performed, showed improvement of recognition accuracy.Among the examples, Example 2 showed best performance.

It was confirmed that accuracy of speech recognition can be improved bythe novel training function that inserts a new layer initialized usingdecomposed parameter sets, is performed. Also, it was confirmed thataccuracy of speech recognition can be improved even when discriminativepre-training that inserts a new layer initialized with random parametersets to the fixed position is performed.

Computer Hardware Component

Referring now to FIG. 7, a schematic of an example of a computer system10, which can be used for the speech recognition system 100, is shown.The computer system 10 shown in FIG. 7 is implemented as computersystem. The computer system 10 is only one example of a suitableprocessing device and is not intended to suggest any limitation as tothe scope of use or functionality of embodiments of the inventiondescribed herein. Regardless, the computer system 10 is capable of beingimplemented and/or performing any of the functionality set forthhereinabove.

The computer system 10 is operational with numerous other generalpurpose or special purpose computing system environments orconfigurations. Examples of well-known computing systems, environments,and/or configurations that may be suitable for use with the computersystem 10 include, but are not limited to, personal computer systems,server computer systems, thin clients, thick clients, hand-held orlaptop devices, in-vehicle devices, multiprocessor systems,microprocessor-based systems, set top boxes, programmable consumerelectronics, network PCs, minicomputer systems, mainframe computersystems, and distributed cloud computing environments that include anyof the above systems or devices, and the like.

The computer system 10 may be described in the general context ofcomputer system-executable instructions, such as program modules, beingexecuted by a computer system. Generally, program modules may includeroutines, programs, objects, components, logic, data structures, and soon that perform particular tasks or implement particular abstract datatypes.

As shown in FIG. 7, the computer system 10 is shown in the form of ageneral-purpose computing device. The components of the computer system10 may include, but are not limited to, a processor (or processingcircuitry) 12 and a memory 16 coupled to the processor 12 by a busincluding a memory bus or memory controller, and a processor or localbus using any of a variety of bus architectures.

The computer system 10 typically includes a variety of computer systemreadable media. Such media may be any available media that is accessibleby the computer system 10, and it includes both volatile andnon-volatile media, removable and non-removable media.

The memory 16 can include computer system readable media in the form ofvolatile memory, such as random access memory (RAM). The computer system10 may further include other removable/non-removable,volatile/non-volatile computer system storage media. By way of exampleonly, the storage system 18 can be provided for reading from and writingto a non-removable, non-volatile magnetic media. As will be furtherdepicted and described below, the storage system 18 may include at leastone program product having a set (e.g., at least one) of program modulesthat are configured to carry out the functions of embodiments of theinvention.

Program/utility, having a set (at least one) of program modules, may bestored in the storage system 18 by way of example, and not limitation,as well as an operating system, one or more application programs, otherprogram modules, and program data. Each of the operating system, one ormore application programs, other program modules, and program data orsome combination thereof, may include an implementation of a networkingenvironment. Program modules generally carry out the functions and/ormethodologies of embodiments of the invention as described herein.

The computer system 10 may also communicate with one or more peripherals24 such as a keyboard, a pointing device, a car navigation system, anaudio system, etc.; a display 26; one or more devices that enable a userto interact with the computer system 10; and/or any devices (e.g.,network card, modem, etc.) that enable the computer system 10 tocommunicate with one or more other computing devices. Such communicationcan occur via Input/Output (I/O) interfaces 22. Still yet, the computersystem 10 can communicate with one or more networks such as a local areanetwork (LAN), a general wide area network (WAN), and/or a publicnetwork (e.g., the Internet) via the network adapter 20. As depicted,the network adapter 20 communicates with the other components of thecomputer system 10 via bus. It should be understood that although notshown, other hardware and/or software components could be used inconjunction with the computer system 10. Examples, include, but are notlimited to: microcode, device drivers, redundant processing units,external disk drive arrays, RAID systems, tape drives, and data archivalstorage systems, etc.

Computer Program Implementation

The present invention may be a computer system, a method, and/or acomputer program product. The computer program product may include acomputer readable storage medium (or media) having computer readableprogram instructions thereon for causing a processor to carry outaspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising”, when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below, if any, areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of one or more aspects of the present inventionhas been presented for purposes of illustration and description but isnot intended to be exhaustive or limited to the invention in the formdisclosed.

Many modifications and variations will be apparent to those of ordinaryskill in the art without departing from the scope and spirit of thedescribed embodiments. The terminology used herein was chosen to bestexplain the principles of the embodiments, the practical application ortechnical improvement over technologies found in the marketplace, or toenable others of ordinary skill in the art to understand the embodimentsdisclosed herein.

What is claimed is:
 1. A computer-implemented method for training aneural network including an input layer, one or more hidden layers, andan output layer, wherein the one or more hidden layers of the neuralnetwork includes one or more fully-connected layers under the outputlayer and one or more convolutional layers between the input layer andthe one or more fully-connected layers, and one layer of the one or morefully-connected layers after a prior pre-training is targeted as the onepre-trained hidden layer for the decomposing, the method comprising:preparing a base of the neural network, the base having at least the onepre-trained hidden layer; decomposing a parameter set into a pluralityof new parameter sets, wherein: the parameter set is represented, atleast partially, as a matrix; the parameter set is associated with theone pre-trained hidden layer in the neural network; the parameter setincludes a bias vector of the one pre-trained hidden layer; anddecomposing the parameter set into the plurality of new parameter setsis based, at least in part, on: applying singular value decomposition tothe matrix to generate a first orthogonal matrix, a singular valuematrix, and a second orthogonal matrix, wherein: the matrix has a weightmatrix connected to the one pre-trained hidden layer; and the pluralityof the new parameter sets is based on the first orthogonal matrix, thesingular value matrix, and the second orthogonal matrix; increasing thenumber of the hidden layers in the neural network based at least in parton the plurality of the new parameter sets, wherein increasing thenumber of hidden layers in the neural network includes: replacing theone pre-trained hidden layer with multiple new layers; setting a firstnew parameter set of the plurality of the new parameter sets to an upperlayer of the multiple new hidden layers, wherein: the first newparameter set includes a first weight matrix obtained from the firstorthogonal matrix and a first bias vector; and setting a second newparameter set of the plurality of the new parameter sets to a lowerlayer of the multiple new hidden layers, wherein: the second newparameter set includes a second weight matrix obtained from the secondorthogonal matrix and a second bias vector, wherein: the first biasvector and the second bias vector are calculated by using the biasvector of the one pre-trained hidden layer; and the first bias vector isset to be identical to the bias vector of the one pre-trained hiddenlayer and the second bias vector is set to be zero; pre-training theneural network based, at least in part, on discriminative pre-training,wherein: the decomposing, the increasing, and the pre-training arerepeatedly performed in a layer-by-layer manner until the number of thehidden layers in the neural network reaches a predetermined number whilethe decomposing is applied to a same position in the neural network; andfine-tuning the neural network after the number of the hidden layers inthe neural network reaches the predetermined number.